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We systematically apply density functional theory to determine the kind of inhomogeneities that 
spontaneously develop in a homogeneous gapless phase of neutral two-flavor superfluid quark matter. 
We consider inhomogeneities in the quark and electron densities and in the phases and amplitude 
of the order parameter. These inhomogeneities are expected to lead the gapless phase to a BCS- 
normal coexisting phase, a Larkin-Ovchinnikov-Fulde-Ferrell (LOFF) state with phase oscillations 
alone, and a LOFF state with amplitude oscillations. We find that which of them the homogeneous 
system tends towards depends sensitively on the chemical potential separation between up and down 
quarks and the gradient energies. 
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£ ' I. INTRODUCTION 

s : 

It has been noted since the seminal work in Refs. 0, Q] that the quark-quark interaction in the color antitriplet 
channel is attractive and drives a Cooper pairing instability in quark matter in the limit of high density. In this 
limit, the ground state for three- flavor quark matter is a homogeneous color- flavor locked state in which all nine 
quarks, associated with three colors and three flavors, are gapped Q. However, color superconducting states, if 
' occurring in compact stars, would not be necessarily homogeneous. This is because a separation of the Fermi surface 
, develops between paired quarks in a realistic situation characterized by nonzero strange quark mass, color and electric 
7— I ' charge neutrality, and weak equilibrium. Once the separation becomes comparable to the gap magnitude, a usual 
BCS state can be energetically unfavorable because of the inevitable increase in the loss of the quark kinetic energy. 
Even in this situation, it is possible to consider other homogeneous paired states such as a gapless state in which 
quark quasiparticles which are gapped in the absence of the Fermi surface separation become gapless 0, 0, @, a 
-««^ ■ color-flavor locked state with condensation of collective modes carrying the same quantum number as mesons |7(, 
' and a paired state with nonspherical Fermi surfaces 8] . Importantly, these homogeneous states are not always stable 
Oh! against inhomogeneities. The most remarkable example is a chromomagnetic instability of the gapless states which 
is characterized by negative Meissner masses squared this signifies that fluctuations in the gluon fields or, 

equivalently, the phases of the order parameter [ll| , develops spontaneously. [Note that in an overall neutral system 
of charged fermions, the gapless states can be stable against homogeneous change in the gap magnitude in contrast to 
the case of neutral Fermi systems (see Ref. |12| for an exception).] So far, many states involving inhomogeneities in 
the quark densities and/or the order parameter 0] have been proposed for an eventual stable state after the onset of 
the chromomagnetic instability, but a systematic energy comparison between them and even a systematic instability 
£3 . analysis with respect to various possible inhomogeneities remain to be performed. 

In this paper, as a first step towards such a systematic analysis, we utilize density functional theory to investigate 
instabilities of a two-flavor homogeneous gapless state with respect to inhomogeneous fluctuations in the quark and 
electron densities and in the phases and amplitude of the order parameter. Here we do not address what the ground 
state is, but all we can clarify is to identify inhomogeneities that grow spontaneously in the homogeneous system, from 
which we can anticipate what kind of inhomogeneous state is likely to be realized. Possibly, the growth of fluctuations 
in the phases and/or amplitude of the order parameter would end up with a Larkin-Ovchinnikov-Fulde-Ferrell (LOFF) 
state 14] in which periodic spatial oscillations occur in the phases and/or amplitude, while the simultaneous growth of 
fluctuations in the density difference between up and down quarks and the gap magnitude could lead to a BCS-normal 
coexisting phase [LSI iLq ] . We take into account finite size corrections such as the energies arising from the density 
and gap gradients and the electrostatic and color Coulomb energies to deduce the structure of the coexisting phase. 
We find that electric charge screening, which is automatically included in the present density functional analysis, is 
crucial to such deduction. The coexisting phase and the LOFF phase with amplitude modulations are related with 
each other in the sense that both phases involve nonzero amplitude and density modulations. As we will see later, it 
is the competition between the density and gap gradient energies that determines which of these the system prefers 
to go to when the system is unstable with respect to fluctuations in the gap magnitude. 

In Sec. [HI we summarize the bulk properties of homogeneous color superconducting states in two-flavor neutral 
quark matter at zero temperature. Section lllll is devoted to constructing density functional theory, by including not 
only the bulk properties summarized in Sec. ^ but also inhomogeneities in the order parameter and the densities of 
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the constituents, and to classifying possible instabilities of the gapless state. In Sec. IIVI we calculate the effective 
potential for the gap magnitude and thereby clarify how the type of instabilities changes with the Fermi surface 
separation and a parameter characterizing the density gradient energy. Our conclusions are given in Sec. We use 
units in which % = c = &b = !• 



II. HOMOGENEOUS PHASES 



In this section we summarize the bulk properties of two-flavor neutral (3 equilibrated quark matter at zero temper- 
ature by using a simple BCS type model. This is instructive for a later stability analysis, which requires the second 
derivatives of the thermodynamic potential density with respect to the phases and magnitude of the order parameter 
and the quark and electron densities. 

We consider uniform ud-flavor quark matter of baryon chemical potential 3/i and zero temperature. We assume 
that the system is in weak equilibrium with the gas of electrons of chemical potential /i e and has zero net color and 
electric charge. We neglect quark and electron masses, and chiral and meson condensates. For a normal state, we 
simply adopt the ideal gas form of the thermodynamic potential density, 

^normal = ^2^2 ^ ; A*i 5 (1) 
i—e.q 

where q = fa stands for the set of color a = R,G, B and flavor / = u,d, and fj, q is the chemical potential of q quarks. 
In this case, (3 equilibrium and charge neutrality lead to 

Hg = fj, - QqUe, (2) 

where Q q is the electric charge of q quarks. The number density of i particles are then given by the corresponding 
chemical potentials fj,i as 

m = A- (3) 

Let us then consider a two-flavor color superconducting (2SC) state in which Cooper pairing occurs in a J p = 0+, 
isoscalar, and -RG-color antitriplet channel. For a pairing interaction, we adopt a contact interaction, 

- G D $f a P£^)®%.P$1>h>v)> (4) 

where P[ b = i^efh^abB-, ^fa is the quark spinor of color a and flavor /, and ifi^ a = ij 2 j°ipj a - Here we assume that 
this pairing interaction is cut off unless the transfer of the relative momentum of paired quarks is smaller than A in 
magnitude. In some sense this cutoff can be considered to simulate the instanton form factor |17| . which allows us to 
solve the gap equation in the same way as the original BCS case in which the cutoff is given by the Debye frequency. 
We take A as 300 MeV, which is small compared with a typical value of the Fermi energy of order 500 MeV. Hereafter 
we will often assume A C A; this is reasonable for a typical range of A of 0-100 MeV. 

Within the mean-field approximation, the pairing gap A is related to the diquark condensate as 

A = 2G D (j fa P^g), (5) 

with the ensemble average (• • •). Taking note of this relation, it is straightforward to write down the thermodynamic 
potential density as Q 



i i A 2 1 r 

i=e,uB,dB i=uR,dR,uG,dG i=uR,dR,uG,dG J t*~ A 

1 - ' 1 - A2 "" Ai E G+-xV (•) 



4 ± \ -4 



where 



12tt 2 ^ n 12tt 2 ^ ^ 4G 2n 2 V 2 A 

i=e,uB,dB i=uR,dR,uG AG i=uR,dR,uG,dG 



_ (J"uR + fJ-dG _ HdR + HuG 



(7) 
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and p is the quasiparticle momentum associated with paired quarks. We have assumed A <C A in the approximate 
estimate. Weak equilibrium and color and electric charge neutrality ensure 

M/a =Ai-<3/Me + Qa = V8, (8) 

where Q" =s = (1/3, 1/3, —2/3) is the color charge for gluon color index a = 8, and /is is the associated color chemical 
potential. The chemical potentials /i e and /Ltg are determined by the neutrality conditions, 

0Q2SC _ dfl 2 sc _ q 

From these conditions, one can show that in the 2SC state, /x e is related to /i and /is as fi e = (3/i — 2/ig)/5, while 
/*s ~ 0.0854/x up to leading order in A/n p. 

We then calculate the gap and the number densities from r22Sc[A;/ii] given by Eq. ©. Variation of f^sc with 
respect to the gap gives rise to the gap equation, 

mhsc A 4/Z 2 A 2A 
° = ^A- = 2G^-^ ln X- (10) 

This is identical with the form of the usual BCS gap equation. We define the gap satisfying Eq. IjlOjl as A = A2S0 
The number densities for i = e, uB, dB are still given by Eq. J3J), while for i = uR, uG, dR, dG we obtain, up to leading 
order in A/ A, 

Obviously, the neutrality constraints (|5J) can be rewritten in terms of rij as 

n e = ^Q / n /a , = ^QrV- (12) 

/a /o 

The 2SC state is relevant only when the separation of the Fermi surface between the u and d quarks, 

^ ~ 2 ~ 2 ~~ 2 ' ^ ^ 

is smaller than the gap A2SC la • This is because the smallest quasiparticle gap associated with paired quarks vanishes 
once the separation reaches A2S0 In fact, according to the usual Bogoliubov method, the corresponding quasiparticle 
energy E_(jp) reads 



E_ (p) = V(P-A) 2 + A 2 - Sfi, (14) 

and for <5/i = A = A2SC0 — 0- Note that in the 2SC state the gap magnitude and thus the condensation energy 

is independent of (S/i except for the slight dependence on 6[i through /2 in Eq. p0(l . This is because quarks forming 
Cooper pairs are located in the middle of the Fermi surfaces as in Eq. J7J, rather than around the respective Fermi 
surfaces. 

For d/i < A2SC, one can find another branch of the homogeneous solution to the gap equation in which the 
gap magnitude is not only smaller than A 2 sc but also the separation 8fi. This state, hereafter referred to as the 
gapless 2SC state, is characterized by the presence of gapless quasiparticle modes associated with paired quarks. 
This is similar to a state originally predicted by Sarma |l8j for superconductors in external magnetic fields. The 
corresponding thermodynamic potential can be described, simply by adding to Eq. © an energy contribution from 
gapless quasiparticle modes of energy E- (p) , as 

1 1 A2 1 /-/Z+A 
'•' u i=uR.dR.iiG.dCrl i - A 



i—e,uB,dB i—uR,dR,uG,dG i—uR,dR,uG,d' 

,+ 



+i E f dpp'E^p), 

i=uR,dR,uG ,dG ^ 



(15) 
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where /i ± = fx ± \j (Sfi) 2 — A 2 . Then, the gap equation reads 



:2SC 



A 4jS 2 A. 2A 



<9A 



+2— 

7T^ 



2 A 



(2/Z 2 - A 2 ) In 



<5/z 



■WW" A 2 



(16) 



By retaining a term of leading order in A//4 in the contribution from the gapless modes, we obtain an approximate 
solution as 

A ~ VA 2 sc(2(5m- A 2 sc), (17) 

which has the same form as that derived by Sarma 0|- Note that the gap magnitude is explicitly dependent on 8pL 
in contrast to the case of the 2SC state. 

The number densities of quarks and electrons can be determined by the derivatives of fi g 2sc with respect to the 
chemical potentials. The number densities for i = e, uB, dB are again given by Eq. J2Jl, while for i = uR, uG, dR, dG 
we obtain, up to leading order in A/ A, 



n, 



3tt 2 

1 
37 2 



p,A 2 2A 



± ^V(^) 2 -A 2 [3/2 2 + (^) 2 



A 



A In 



W(W - a 2 



(18) 



where the upper and lower sign in the right side are taken for d quarks and u quarks, respectively. 

It is instructive to note that the gapless state is stable against small homogeneous change in the gap magnitude 
in charged Fermi systems because of the neutrality constraints 5] , although it is generally unstable in neutral Fermi 
systems. (In fact, without charge, at given Sfj, below A2SCJ the gap magnitude and thus the condensation energy 
would be larger for the 2SC state than for the gapless state.) By substituting the above-obtained densities into Eq. 
(|12fl . one obtains an approximate relation between A and 5/j, as 



A 



(W - 



(jj, - 45fi/3) 3 ~ (2W 
6(pt — V/3) 2 



(19) 



Here we have assumed (i$ = 0, which is known to be a reasonable approximation since nonzero us can arise solely 
from nonzero A. By taking note of the condition that Eqs. (|17fl and i|19fl are compatible,we can find that the gapless 
state is energetically favored over the normal and 2SC states at intermediate coupling 0. In fact, the condition for 
the presence of the gapless and neutral solution approximately reads 



0.11/x < Sn < —n 



(20) 



(See Fig. 1 for an example of the gapless and neutral solution.) By combining this relation with the condition, 
A2sc/2 ^ S/i < A 2 sc, for the presence of the gapless 2SC solution, we obtain 



0.22/i < A 2S c < -^H- 



(21) 



This relation implies that above (or below) this region of A 2 sc > the neutral solution, that is, the intersection of the 
neutral constraint as shown by the dotted line in Fig. 1 and the solution of the gap equation as shown by the solid 
line (or the line of A = 0) in Fig. 1, lies only in the region of the 2SC (or normal) phase 



III. INSTABILITY WITH RESPECT TO INHOMOGENEITIES 



As mentioned in the previous section, the gapless 2SC state is stable against small homogeneous change in the gap 
magnitude. However, it is necessary to examine the stability against inhomogeneities. In fact, it is known that the 
gapless 2SC state is unstable with respect to spontaneous generation of gauge fields . In this section, we construct 
density functional theory to analyze the stability against inhomogeneities, and then describe typical instabilities which 
might occur in the gapless 2SC state and even in the 2SC state. 
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FIG. 1: The homogeneous gap A as a function of S/j, obtained for /i = 500 MeV and A2sc(5m = 0) = 120 MeV. The upper 
solid line is the 2SC solution, and the lower solid line is the gapless 2SC (g2SC) solution [see Eq. Ijl7^ ]. while the dotted line 
denotes the neutrality constraint [see Eq. The intersection of the g2SC solution and the dotted line corresponds to 

the homogeneous neutral solution to the gap equation. Note that the real 2SC solution is dependent on Sfi through the Sfi 
dependence of fl, although such a dependence is negligible under the approximation utilized here Q. 



A. Density functional theory 



We proceed to write the energy density functional allowing for infinitesimal inhomogencitics in the phases and 
magnitude of the order parameter and in the quark and electron densities, i.e., the position dependent densities 
rii(r) = rii + 5rii(r), gap magnitude A(r) = A + SA(r), electromagnetic field A(r), and gluon fields A°(r) (as we 
shall see below, any nonzero gauge fields including constant ones correspond to inhomogeneities in the phases of the 
order parameter). For inhomogeneities of spatial scale larger than the coherence length, it is sufficient to adopt the 
local density approximation for the bulk energy and keep the energies arising from the gap and density gradients up 
to leading order. Thus, the total thermodynamic potential for given /i reads 



ntot= / d 3 r r! (g)2SC (n 4 (r),A(r))+ / d 3 r n g {n i (r),A(r),A(r),A a (r))+E c +J2 E C- 

a 

Here f^sc an d ^ g 2SC ar e given by Eqs. JBJ) and (|15|) . 

A" 



fhab 



+2i4 0) 



E 

aba 

E 

aba 



i 9 ^A a +i5 ab -A 



S bB A 



S nB A ( (UV - ig^A a + id ab -A 



5 bB A 



is the gradient energy, 



E C = / d 3 n j d 3 r 2 



p(rx)p(r 2 ) 
ki - r 2 \ 



(22) 



(23) 



(24) 
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with 



P = e ( E Qf n fa 
fa 



is the electrostatic Coulomb energy, and 



p a ( ri )p«(r 2 ) 



ri - r 2 



with 



(25) 



(26) 



(27) 



fab 



is the color Coulomb energy. Here we have ignored the gluon contribution to the color density since it is of higher 
order in A°(r). For the covariant derivative, we have taken the form relevant for the gap which has a B direction 
in anti-color space, and e/3 corresponds to the electric charge carried by a Cooper pair. The local thermodynamic 
potential, up to second order in various inhomogencitics, can then be written as 



n 



tot 



r> 

f =- [ d 3 r 



E 



__^(r)^(r) + 2E 1 rg 5 -^W«A(r) + — ^(<5A(r)) 



4/- 

4E / ^ 2 %a^n /a (q)K lfc (q)+2(4° ) +A 2 4 1) ) / 7S3 ^ A(q)| 2 

+ ^E / ^(mSf) 2 A a (r)A^(r) + ^ / d 3 r(m^) 2 A«(r)A(r) + \ f d\{m M f{A{v)f 

+ 2j (2^?^ (q)l + 2^IW) 



d3q V(q)l 2 , 



(28) 



where fio is the thermodynamic potential of the homogeneous state, 5rij(q), <5A(q), p(q), and p"(q) are the Fourier 
transforms of the corresponding inhomogencitics, and 



0, 



a = 1-3 
a = 4-7 



Ki^+A^VA 2 , a -8, 



(29) 



(™m) 2 = ^^(4 0) + A 2 i4!VA 2 , (30) 

and 

(m M ) 2 = ^(4 0) +A 2 ^)e 2 A 2 (31) 

are the Meissner masses squared for gluon-gluon, gluon-photon, and photon-photon channels, respectively. Here we 
note that the first order terms vanish due to equilibrium, i.e., 

d°(g)2SC _ d^(g)2SC _ 

and that the terms including both A or A" and SA or Srii do not appear up to second order. We also neglect the 
electron contribution to the gradient energy, which is expected to be small compared with the quark contribution 
that will be discussed below. 

We can simplify the color Coulomb energy by only retaining the components of a = 3,8. In fact, for a ^ 3,8, 
p a turns out to be zero; any deviation of p a from zero for a ^ 3,8 would result in energy increase because there 
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is no dependence on p a in the other energy terms. (Vanishing p a conforms to the color neutrality J d 3 rp a (r) = 0.) 

3n as 

d 3 q ( 



Consequently, the color Coulomb energy can be written as the sum of -Eg -3 and -Eg -8 given by 



e; 



ic*=3,8 
C 



(27T) 



(33) 



where 5n a = J2f Sn fa and Q%~ 3 = (l/\/3, -l/\/3,0). 

For later use, we write the expressions for the second derivatives of the thermodynamic potential density as 



i (g)2SC 



4/2 2 



OA 2 



1 -9(6(1- A) 



5/i 



V(^) 2 - A 2 ft 



A 2 ln y^F 



A 2 + 5/Li 



(34) 



<9 2 ^(g)2SC _ 



dntdA 



dA 



A 

A 
^0, 



■ln 



2A 



+ 0(<5/x - A) - In 



2A 



- ln — + 9(6(i - A) - In 



d 2 n 



(g)2SC 



dpi 

dn.j 



ft 



2/2 2 

2fL 2 

6ij 
0, 




9(6(i- A) 



1 - 9(6(i- A) 



5(i 



^(5(i) 2 -A 2 / 
6(i 

V(S(i) 2 -A 2 / 



TV 

3n,i 



2/3 



, i = uR, uG 

, i — dR, dG 
i = uB, dB, e, 



for (i,j) = (ua,ub), (da,db) 

for (i,j) = (ua,db),(da,ub) 

for i = uB, dB, e 
for others, 



(35) 



(36) 



where a = R,G and b = R,G. In Eqs. I|35|l and we have ignored higher order terms with respect to A/ A and 
A//x. In evaluating d(ii/dA we made use of drii/dA\ ni = in Eqs. and (|18|l . Note that in the 2SC state, dfii/drij 
diverge for the combinations of (i,j) participating in Cooper pairing. This feature, which comes from the fact that the 
corresponding part of the thermodynamic potential JSJl does not depend explicitly on 5p, suggests that the number 
densities for i = uR, uG, dR, dG would not change in the 2SC state. 

We now specify the parameters Bf a ^b, Kj ■ > and K^p characterizing the gradient e nerg y. To a first approximation, 
the density gradient term can be estimated as the Weizsacker correction term Ey/ which is the leading-order 
quantum correction to the Thomas-Fermi model for the energy of an inhomogeneous ideal Fermi gas. By following an 
argument for the nonrelativistic case |l9j |. it is straightforward to obtain the Weizsacker term for an ultrarelativistic 
system as considered here as 

1 ^i + 21n- 2/1/o ^ |Vri/o ' 2 



E 



w 



E 

fa 



d 6 r 



72(37r 2 )V3 



mj 



x 4/3 



(37) 



where we have introduced nonzero quark masses m/ of order 5 MeV in order to obtain a finite result. The corresponding 



values of Bf a ^b can be estimated as Bf a ^ = 5 a b6fhBj a with Bj a 



]J a ~ 50(0.5 fixT 3 /n /a ) 4 / 3 MeV fm 5 . However, effects 
of the interaction between quarks can significantly modify the Weizsacker term. This is expected from the extended 
Thomas-Fermi model for atomic nuclei in which the interaction effects induce the terms including the difference and 
the sum of the proton and neutron density gradients and to increase the overall magnitude of the gradient term by a 
factor of 3-4. Considering uncertainties due to these interaction effects, we take 

Bfa.hb = 5 a b6fhBY a + B,s + BA(35 a b — 1), (38) 

with unknown parameters, Bf n , B g, and Ba- (Even the Weizsacker correction term can be modified by the interaction 
effects in a relativistic system |2(|.) This choice leads to 



\ E B fa,hb(Vn fa ) ■ (Vn hb ) = ~X)B%|Wn /o | 

fhab fa 



1 -B S 

2 6 



E 



\76n a 



1 



B A (\V6n R - V5n G \ 2 + \WSn G - V5n B \ 2 + \V6n B - V5n R \ 



(39) 
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Note that the present choice of Bf a ^b allows for the different interaction strengths for the color symmetric and 
antisymmetric combinations between a and b, as expected from the gluon exchange interactions. 

For^ 0) andi4 1} , 

we use the known weak coupling expressions as functions of A, fi, and 5/i 0|- These expressions 

read 



K, 



(o) _ fi 



6tt 2 A 2 



1 - 2 



26 fiy/ (6 fi) 2 - A 1 

A 2 



6(5[i - A) 



(40) 



and 



K, 



(i) 



12tt 2 A 4 



: A 2 



/ 4<5 A i V /(<5^) 2 - A 2 



A 2 



(41) 



Combining Eqs. I|4l)[) and (|41|) with Eqs. I|29|) - (|31[l . one can reproduce the Meissner masses in the weak coupling limit 
9] . It is interesting to note that near the transition temperature, where the gap magnitude is suppressed, the stiffness 



parameter is still similar to expression (|40|l in magnitude, while the term associated with Kj^ becomes negligible as 
compared with the term associated with ■ 



B. Chromomagnetic instability 

From the energy density functional (|28|l . we now proceed to consider possible instabilities in the 2SC and gapless 
2SC phases. We first examine chromomagnetic instabilities predicted to occur in both phases Q- The chromomagnetic 
instability is an instability associated with the phase inhomogeneities or, equivalently, A Q (r) and A(r). We set the 
other inhomogeneities to be zero. Then, the sign of the Meissner masses squared or, equivalently, the sign of the 
specific combination of the stiffness parameters as in Eqs. I|29[l - H31|l . plays a role in determining the occurrence of the 
instability. We thus have only to take note of the stiffness parameters, which is similar to the way Giannakis and Ren 
[Tl| approached the chromomagnetic instability associated with the £7(1) phase. 

The instability related to negative (m^f) 2 , ( m %i) 2 an d (toj\/) 2 occurs only in the gapless 2SC phase (Sfi > A), 
as can be seen from the expressions for and K^r. This instability can be viewed as the tendency towards a 

plane- wave LOFF state in which the gap spatially oscillates like e zqr (phase oscillations). This is because the a = 8 
component of the 5£7(3) phase acts like a usual £7(1) phase for the RG pairing considered here for the 2SC state. 

In order to look further into the relation between the SU(3) and £7(1) phases, we explicitly write down the su- 
percurrents associated with the gradient of the £7(1) phase <f> and the color SU(3) phases 4> a . A degenerate order 
parameter set of the 2SC states can be obtained by transforming the 2SC state specified in the previous section under 
global £7(1) and color rotation as 

d a =(e i *e i ^o i ^* a ) A. (42) 

V / aB 

By inserting this order parameter into S a B^ in the gradient energy l\2'M one can obtain the supercurrents from pi) 

Jl ~ BAi' 4 ~ BAf [ 6) 



The results read 




a = 1-3 

K^gA 2 d l( f> Q - (m%?) 2 Af , a = 4-7 

-(4 0) + A 2 K ( T %A 2 (Afit - - Kf )Uf - (m^fAi, a = 8, 



(44) 



and 



Ji = ^Jf- (45) 



It is thus evident that the gauge fields play the role of the phase gradients in the sense that any supercurrent pattern 
created by the phase gradients can be reproduced by the gauge fields alone. 
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From these supercurrents, we can find that the system, if undergoing the chromomagnetic instabilities associated 
with A and/or A 8 , would tend to a plane-wave LOFF state. In fact, the proportionality of J to J 8 implies that any 
inhomogeneous state created by A 8 could be created by A. It is thus sufficient to consider spontaneous generation 
of A. As long as A is small and uniform, the resultant supercurrent can be reproduced by 0(r) = q • r with q = |A, 
which in turn characterizes a plane wave LOFF state. 

There is another type of chromomagnetic instability, which is associated with negative (mj^) 2 , (m^|) 2 , (m^f) 2 , and 
(m^) 2 . This instability appears not only in the gapless 2SC state but also in a part of the 2SC state that fulfills 
dfi > A/y2- This instability couples with the phase gradients of ^4-^7. For small and uniform A a with a = 4-7, 
the state can be described by <fi a (r) — q • r with q = —gA a . Consequently, this state corresponds to a LOFF state 
characterized by the color rotation exp(zA a q • r/2) of the gap. Note, however, that in the gapless 2SC state, the 
absolute values of (m\j) 2 , • • • , (mj^) 2 are not larger than those of (m 88 ) 2 , (m\ { ) 2 , and (ttim) 2 [see Eqs. lj2~9"|l -lpni )l with 
Eqs. (|40|l and (|41|l ]. We can thus conclude that in the gapless 2SC state, the tendency to the LOFF state with color 
rotations is weaker than that with U(l) rotations. 

Note that the instabilities considered here do not involve any variation of the gap amplitude or the particle densities. 
We will thus consider the cases in which such variations occur in the next two subsections and discuss the possibility 
of instabilities with respect to them. 

C. Gap amplitude instability 

We now consider inhomogeneities in the gap amplitude by setting the other inhomogeneities to be zero. We can 
see such an instability from the second derivative of f2( g )2SC with respect to <5A(q). Note that the second derivative 
(|34|l . which is relevant for q = 0, can be negative only in the gapless SC state, implying that this comes from the 
presence of the gapless modes. The associated instability remains in the limit of q — * in gapless neutral superfluids 
|l8| which are generally unstable with respect to homogeneous (q = 0) change in the gap amplitude, while neutrality 
constraints keep the gapless charged superconductors stable against homogeneous change in the gap magnitude. This 
is because this change is accompanied by homogeneous change in the quark and electron densities, leading to increase 
in the total kinetic energy. 

It is important to note that at nonzero q, the system becomes more unstable by having the energy lowered by the 
gradient energy term associated with inhomogeneities in the gap amplitude in the gapless phase where the stiffness 
parameters are negative. Then the neutrality constraints no longer hinder the system from undergoing the instability 
at nonzero q. This implies the tendency towards a LOFF state in which the gap amplitude oscillates. In general, the 
gap amplitude instability develops in the gapless state together with the chromomagnetic instability, though we will 
refer to this situation simply as the instability with respect to the gap amplitude. 

D. Stability against density fluctuations 

We finally consider the case in which inhomogeneities are present only in the quark and electron densities. The 
possible occurrence of instabilities depends on the structure of <9 2 f2( g ) 2 sc/ 'drudrij. In order for the system to be stable 
against small density modulations, all eigenvalues of this hermitian matrix must be positive, i.e., any minors of the 
determinant must be positive. This condition holds both for the 2SC state and for the gapless 2SC state. 

It is nonetheless important to note that the Fermi surface separation between d and u quarks can vary nonuniformly 
in the gapless 2SC state, which is driven by the gap amplitude instability discussed above in such a way that as 
the gap amplitude increases, the Fermi surface separation decreases. Generally, such density variation produces a 
positive gradient energy. If this energy dominates over the gradient energy associated with inhomogeneities in the 
gap amplitude, one can expect that the system tends to a coexisitng phase of large BCS (2SC) domains with small 
Fermi surface separation and large normal domains with large separation ^(|. This phase can be viewed as a state 
in which clustering of d quarks occurs within normal domains. 

In order to clarify what kind of structure this clustering takes on, one needs to take full account of inhomogeneities 
in the densities and the gap amplitude; this account would automatically involve charge screening, which plays 
an important role in determining the structure. This structure lies in between the strong screening limit (phase 
separation) and the screeningless limit, as will be quantified in the next section. We remark that the same kind of 
phase separation was recently observed in a superfluid gas of atomic fermions with unequal numbers of two components 
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IV. EFFECTIVE POTENTIAL FOR THE GAP MAGNITUDE 



In this section we systematically consider inhomogeneities in the quark and electron densities and in the phases 
and amplitude of the order parameter to determine the kind of inhomogeneities that spontaneously develop in the 
2SC and gapless 2SC states. In doing so, we first focus on the bulk (q independent) part of the effective potential 
for the variation of the gap amplitude. We then include the effect of the Coulomb and gradient parts in the effective 
potential and thereby clarify what state the system tends to. 

Let us begin with the expression for the effective potential v(g) for given <5A(q), which can be obtained from the 
local thermodynamic potential (|28[) as 



^tot — &o 
1 

+ 2 



d 3 q 
(2tt) s 



v(q)\SA(q)f 



lj2 J dM<f) 2 A Q (r)A^(r) + i]T / d 3 r(m^) 2 A"(r)A(r) + \ J d 3 r(m M ) 2 (A(r)) 5 

a/3 a 



where 



v(q) = v + 4(i4 0) + A 2 4V + E WMSfh 

fhab 
|2 



B S + B A (36 ab 1)] |JA(q)|2 Q 



E f aQftnfa(q)-5n e ( q )\ + (VSg/2) 2 J2 a =3,8 IE Q%& n a(q)\ i 



|<SA(q)f 



with the bulk contribution 

d 2 n 



(g)2SC 



OA 2 



E 



5n t (q) 5n*(q) 



SA(q) SA*(q) 



d 2 n 



(g)2SC 



dAdrii 



E 



fa a (q)(kt*(q) d 2 fl {g)2SC 
\SA(q)\ 2 driidrij ' 



(46) 



(47) 



(48) 



From the condition that this effective potential takes on a minimal value, i.e., <9f2 to t/ 'dSn^q) = 0, we acquire the 
relations 



and 



^Q/fc/a(q)-K(q) 

fa. 



(49) 



yZS^-Sn^q) 
^ on fa 



Q f e 2 



E QhSn h b(q) - Sn e (q) 



lib 



'yV 
2q 



E Q a aY,^n hb {q) 



-q 2 Y^[BYJa b 5 f h + B S + B A (3S ab - l)]Sn hb (q) - -j^6A(q). 



a=3,8 hb 
d^fa 



hb 



(50) 



In the following, to a first approximation, we search for the solution to Eqs. (|49|l and (|50|l in the absence of finite 
size corrections, namely, in the absence of the terms coming from the gradient and Coulomb energies. This solution 
is good enough to control the sign of v(q). We then extend the calculations to the case with finite size corrections to 
clarify the detailed spatial structure of the instabilities of the system as function of S/i. 



A. Case without finite size corrections 



In Sees. IIII ri and lTTl Dl we have found that the gapless state is unstable with respect to the gap amplitude oscillations 
and stable with respect to density fluctuations by considering <5A(q) and Srii(q) separately. The bulk part, vo, of the 
effective potential, which we will focus on in this subsection, is expected to clarify how 5rii(q) is related with i5A(q). 

Once one neglects the Coulomb and gradient parts of the effective potential, one is allowed by the symmetry of v 
in color and flavor space to set 

E] 5n ua = <5n_ + 6n + , E] & n da = —Sri- + 6n + . (51) 

a=R,G a=R,G 
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Then, one can obtain the relation between <5A(q) and 5n,(q) from Eqs. I|49(l and l|50|) . by retaining the term of leading 
order in Sfi/fi for d/Xi/dA, Eq. as 



<5n_ = 5n + = 0, <5ri e = 5n U B = Sn c iB = 0, 



for the 2SC state, and as 



<5n_ = 



fL 2 6A 



<5n+ = 0, 



(52) 



(53) 



8n e 



Sn uB = 5n dB = 0, 



(54) 



for the gapless 2SC state. 

The neglect of the gradient and Coulomb energies is a good approximation as long as the typical scale of the 
incompressibilities d^i/drij, which is of order 7r 2 //i 2 , is sufficiently large. However, it is important to note that the 
color Coulomb energies can be comparable to or even greater than the bulk contribution. It is thus reasonable to keep 



the color Coulomb energies vanishingly small. Due to 5n + = 0, E£~ 8 = 0, while, from Eq. (|33|l . 

Sn U R + Sn dR = Sn uG + Sn dG . 
By substituting the resulting 5n,i into vq, we obtain 



Eg 



=3 _ 



leads to 



(55) 



v = 



Aft 2 



i - e(Sfi - a) 



Sfj, 



V(^) 2 - A 2 



1 



A 2 



A 2 ^(Sli) 2 - A 2 

— ln x — 



(56) 



The sign of vo predominantly determines the sign of v(q), as we will numerically confirm later. Note that vo is positive 
(negative) in the 2SC (gapless 2SC) state. This implies that the gapless 2SC state tends to be unstable with respect 
to small variations of the gap amplitude, while the 2SC state being stable. Consequently the 2SC state is unstable 
only to the variation of the phases of a — 4-7; this instability occurs when S/j, > A/^/2 as shown in Sec. 1111 Hi 

Note that the gap amplitude instability, which is predicted to occur in the gapless 2SC state, results in the d quark 
clustering. In fact, the density difference between u and d quarks becomes smaller with increasing (5A(q), as can be 
seen from Eq. (|53|l . 



B. Case with finite size corrections 



We now proceed to take into account finite size corrections due to the Coulomb and gradient energies in evaluating 
the effective potential. As we shall see, the sign of the gradient contribution and the effect of electric charge screening 
play a role in determining the configuration of inhomogeneities that develop spontaneously. 

By substituting the bulk solutions i|52fl - H54[) into Eq. I|47|l . we obtain the gradient contribution to the effective 
potential as 

^ 2 _ A /^(0) , A 2 7^(1)^2 , roW'r c , o , R /or - | fa /a (q)fa^(q) 2 
v g q = 4(A T + A K T )q + 2_^[^ fa d abOfh + B s + B A {6d ab - 1)\ [^7^12 q 



fhab 



3tt 2 A 2 



1 - 0(6(1 - A) 



V(^) 2 A 2 



B%0(5n-A) 

f=u,d, a—R,G 



A/2 2 



2tt 2 V(^) 2 - A 2 



>g 2 . (57) 



Here, for the gapless 2SC state, we have used 



&n uR = 5n uG = -5n dR = -8n dG = -5n. 



(58) 



This relation can be obtained by combining Eq. (|55|l with Eq. I|53|l and by minimizing the Weizsacker term under 
^ufl = -^mG an d — -^dG: which stem from the relations, n U R = n uG and n d R — n dG , given by Eq. (|18(l as well 
as the relations, [i u r = fi uG and fi d R = n dG , given by Eq. JSJl. We remark that the density gradient contributions 
associated with the parameters B$ and Ba vanish in the absence of Snu :Gt B- 

From v g we examine how instabilities in the gap magnitude develop in the gapless 2SC state. For p, ~ 500 MeV, 
A ~ 100 MeV, and B^ a ~ 50 MeV fm 5 , v g is negative, due to the negative gap gradient energy, except in the 
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immediate vicinity of the gapless onset <5/i = A, i.e., v g takes a positive value only for Sfi/A < x-i ~ 1.011. When v g 
is negative, the system tends to a LOFF state with amplitude oscillations since a larger q is more favorable. In this 
LOFF state, the amplitude oscillation is allowed to occur at a spatial scale of the order of the coherence length. Note 
that the local thermodynamic potential l|46l) suggests an even stronger instability of the system due to the variation 
of the gauge fields, which is independent of SA because the terms like <9 2 f2( g ) 2 sc/<9 j 4? 9 A vanish up to second order 
in the inhomogeneities. Since this additional instability is particularly strong for A and/or A 8 as characterized by 
the negatively divergent behavior of (to 88 ) 2 , (to 8 ^) 2 , and (tom) 2 near the onset of the gapless 2SC state, the eventual 
LOFF state would presumably be described as a superposition of plane waves involving the appreciable change in the 
amplitude. Note, however, that this state still undergoes chromomagnetic instabilities associated with a = 4-7 [23| . 
In any case, we shall refer to this state with v g < (large q 2 ) as the LOFF state with amplitude oscillations. 

It is instructive to consider the case in which v g is positive although this is limited to the immediate vicinity of 
Sfx = A for the typical case in which p, ~ 500 MeV, A ~ 100 MeV, and Bj a ~ 50 MeV fm 5 . This situation implies 
the tendency to either a BCS-normal mixed state or phase separated state. As we mentioned before, the mixed or 
phase separated state is not clearly separable from the LOFF state with amplitude oscillations. Qualitatively, if q 2 is 
large as suggested by v g < then the resultant inhomogeneity is characteristic of the amplitude LOFF state. There 
comes out the mixed phase next as q 2 decreases and finally the phase separated state for even smaller q 2 . Let us 
now quantify this criterion by determining the characteristic scale of q 2 — Q 2 that distinguishes between the mixed 
and phase separated states. The typical scale of the spatial structure of the mixed phase, 2irQ~ 1 , is determined in 
such a way as to minimize the sum of the gradient and Coulomb contributions to the effective potential. This scale 
arises because the gradient energy increases with q while the Coulomb energy decreases with q. In evaluating the 
Coulomb contribution, it is important to allow for the variation of the electron density, which acts to screen the 
charge distribution formed by the R and G quark components. The Coulomb contribution including these screening 
corrections can be expressed as 



v c 



2 

J2 fa Qf Sn fa(q) - Sn e (q) 



dn e \Sn e (q)\ 2 

dn e \SA(q)\ 2 \5A{q)\ 2 q 2 

e 2 \5n-{g)\ 2 
(q 2 + k 2 TF )\5 A(q)\ 2 > 



(59) 



where 



fc TF = — (60) 



is the inverse of the Thomas- Fermi screening length. Here we have used Eq. (|49(l in evaluating Sn e , and we have ignored 
a possible screening by quarks, since it would lead to a nonvanishing net color charge which should be energetically 
disfavored. 

By combining Eqs. Q57JI and 1)59(1 . we can estimate the value of q = Q that minimizes the sum of v g q 2 and Vc as 



2 \ 1/2 



Q = -*4f + ( jj . (6i) 

where f3 g = v g (\SA\/\5n^\) 2 . At q = Q, the sum reads 

v g Q 2 +v c = {2^3^ - /? 9 fc 2 F ) Jjj^j£. (62) 

When the value of (3 g is sufficiently small that Q 2 > fc^ F , it is expected that the normal-BCS mixed state occurs 
in the form of a Coulomb lattice of periodicity of order 27rQ _1 . We remark that a similar situation occurs in the 
liquid-gas mixed phase of nuclear matter at subnuclear densities |24j . 

Near the gapless onset <5/i/A = 1 we can expand Q 2 with respect to e = — las 

W TF VW 6(£^)3/2A2/2 2 + 12(B^)5/2A 4 /1 4 + ' { ' 

where we denote Bf a simply by B w . In the typical case in which A = 100 MeV and B w = = 50 MeV fm 5 , 



'fa 

near the gapless onset, we obtain Q 2 ~ e/V B w ~ 0.6 fm~ 2 ^> fc^ F ~ 0.01 fm~ 2 . In this case, the normal-BCS mixed 
phase is expected in the form of a Coulomb lattice of periodicity ~ 2n/Q ~ 8 fm. We plot Q 2 for this case in Fig. 2(a) 
and it is clear from the figure that Q 2 is much larger than k^ F in the whole region up to 8/i/A = 
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FIG. 2: Spatial momentum squared Q 2 at which the system is most unstable to the gap amplitude variation. The solid lines 
are the results from Eq. I|fil|l . and the dashed lines are the results from Eq. I|63|l . (a) The typical case A = 100 MeV and 
B w — = 50 MeV fm 5 . For 1 < Sfi/A < X2 — 1.011, Q 2 is much larger than fc^p indicating that the system takes a form 
of the Coulomb lattice with periodicity ~ 2-r/Q. (b) For large B = 30B^ there appears the region in which Q 2 is smaller 
than fc^p. Then the screening effect is so significant that the system can be phase separated. 



The value of Q 2 , as seen from Eq. I|63|l. depends strongly on the value of B w . For sufficiently large values of B w , 
we obtain a negative value of Q 2 . This indicates that the real solution is Q = and that the electron screening is 
sufficiently strong to separate the system into the normal and BCS (2SC) regions that are locally charge neutral. This 
situation is expected to continue until Q 2 amounts to ~ k^ F . Near Q — fcxF, at which we define x\ = 5[i/A, the 
electron screening ceases to ensure the local charge neutrality. We have calculated Q 2 for the case of B w — 30B^ 
and depicted the behavior of Q 2 in Fig. 2(b). In this case, the decreasing behavior of Q 2 is dominated by the term, 
— fc^ F , in Eq. We also note that in this case x-i does not exist since v g is positive for S/i/A > 1. In fact, v g can 

be negative only when 

BW < s B <- (64> 

where amounts to - 200 MeV fm 5 for A - 100 MeV and p, ~ 500 MeV. 

Recall that the local thermodynamic potential l|46|) suggests an even stronger instability due to the variation of the 
gauge fields. This implies the possibility that the normal state in the mixed phase is replaced by the LOFF state 
without amplitude oscillations. Near 8[i = A, \Sn-/SA\ is far larger than the typical scale fi 2 /tt 2 of the density of 
states, indicating that the gap might remain in the region where the Fermi surface separation between u and d quarks 
is maximal. Consequently, the LOFF state, which occurs only for nonzero A, could appear in such a region. 

In Fig. 3, we summarize the instabilities of the homogeneous system with respect to various inhomogeneities and 
the resultant possible states on the B w versus 8fi/A plane. Through the last term in the thermodynamic potential 
IjlSp. the presence of the gapless modes in the gapless 2SC state underlies all the instabilities but the chromomagnetic 
instability that occurs also in the 2SC state. It is noteworthy that no LOFF state with amplitude oscillations is 
expected in the gapless state when B w is larger than the critical value B^f . We emphasize that all we can know from 
the present stability analysis is how the system is driven by infinitesimal variations of the densities and the order 
parameter. This property, while being suggestive of the final destination of the unstable system, does not tell us what 
the ground state really is (see Ref. for its candidates). 

The results shown in Fig. 3 are in agreement with our intuitive expectation: For larger B w the BCS-normal interface 
costs a larger energy, meaning that a phase consisting of larger domains is preferable energetically. Therefore the region 
of the BCS-normal phase separation becomes wider on the phase diagram with increasing B w . It should be mentioned 
that within the present approximation including the energy variations of second order in the inhomogeneities, there is 
no clear transition between the mixed and separated states; there can be a first order transition between them, whose 
clarification is beyond the scope of this paper. 



14 



LOFF 
with 

amplitude 
oscillations 



BCS-normal 
phase separation 



BCS-normal 
mixed phase 



1/V2 



LOFF without amplitude oscillations 



Stable 2SC 



10 1 



b w ibZ 



io z 



FIG. 3: The possible phase structures implied by a stability analysis on the B versus dfx/A plane. The parameter B w is 
normalized by a typical value Bq — 50 MeV fm 5 . The solid line slightly above 5/i/A = 1 represents v(Q) = 0, while Sfi/A = 1 
corresponds to Vo = 0. It is thus obvious that the bulk part vo given by Eq. 156H dominates the behavior of v(Q). The line 
labeled with X2 is determined by the condition v g = from Eq. 15711 . The dashed line labeled with x\ indicates the criterion 
Q 2 = fcxFi m the left (right) of which the system tends towards a mixed (separated) state. Note that with increasing S/i/A, X2 
and X\ asymptotically approach a critical line, B w = B]7, given by Eq. 1641 from left and right, respectively. 



Note that we changed B w as a variable parameter, while it is equivalent to changing A. This is because the 
conditions determining x\ and X2 would not be altered if A is multiplied by an arbitrary factor £ and at the same 
time B w is divided by £ 2 . Thus, a larger B w with a fixed A corresponds to a larger A with a fixed B w . Since A 
should not exceed A 2 sc (see Fig. 1), however, drastic change in A would not be realistic. 

We conclude this section by noting that the density functional was originally written by assuming that q 
is small. If the true value of q is large, therefore, one would need extension of the framework by including the 
higher-order gradient energies. One encounters this situation especially when S/J./A is close to x-i as can be seen in 
Fig. 2. 



V. CONCLUSIONS 



We have systematically examined instabilities of the homogeneous and neutral superconducting states for two-flavor 
quark matter with respect to inhomogeneities in the quark and electron densities and in the phases and amplitude of 
the order parameter. The result was summarized in Fig. 3. We thus clarified the role of the gapless quark modes, the 
density and gap gradients, and the electron screening in determining the structure of spontaneous fluctuations. 

However, open problems still remain. First, the spatial scale of the possible LOFF states remains to be estimated. 
This is because this scale is determined by the competition between the negative second-order gradient term and the 
unknown fourth-order term [25| . Extension to the case of nonzero temperature is significant for possible application 
to the interiors of compact stars. Since the influence of the gapless quark modes on the thermodynamic potential is 
smoothed out at nonzero temperature, the instabilities as discussed here are likely to be weakened. Finally, extension 
to the case of three-flavor quark matter is also important for the description of a more realistic situation. For this 
purpose, the mean-field analysis of the phase diagram and the chromomagnetic instabilities would be a good starting 
point. 
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